Imaging apparatus

ABSTRACT

The cost and power consumption of an imaging apparatus are reduced by facilitating detection of an incident angle of a light beam transmitted through a grating substrate. An image sensor converts an optical image captured by pixels arranged on an imaging surface and outputs the converted image signal. A modulator is configured to modulate intensity of light; and an image processing circuit performs image processing of the output image signal. The modulator has a grating substrate, a grating pattern formed on a back surface side of the grating substrate arranged in proximity to the light receiving surface of the image sensor; and a grating pattern formed on a front surface facing the back surface. Each of the grating patterns is constituted of a plurality of concentric circles. The modulator performs intensity modulation on the light transmitted through the grating pattern and outputs the modulated light to the image sensor.

TECHNICAL FIELD

The present invention relates to an imaging apparatus, and particularlyrelates to a technique effective for enhancing performance of theimaging apparatus.

BACKGROUND ART

There is a demand for a thinner digital camera to be mounted on asmartphone or the like. Examples of the technique for realizing thistype of thinner digital camera include a technique that obtains anobject image without using a lens (see, for example, Patent Document 1).

This technique obtains an image of an external object by attaching aspecial diffraction grating substrate to an image sensor and obtainingan incident angle of incident light by the inverse problem calculationon the basis of a projection pattern generated on the image sensor bylight transmitted through the diffraction grating substrate.

RELATED ART DOCUMENTS Patent Documents

Patent Document 1: US Patent Application Publication No. 2014/0253781

SUMMARY OF THE INVENTION Problems to be Solved by the Invention

In the above-described Patent Document 1, the diffraction gratingpattern formed on an upper surface of the substrate to be attached tothe image sensor is a special grating pattern such as a spiral pattern,and an image of an object is obtained by solving an inverse problem forreproducing the image based on the projection pattern received by theimage sensor. However, there is a problem of complication of calculationfor solving the inverse problem.

If the calculation is complicated, the processing time naturally becomeslong, so that it takes a long time to display the image. Ahigh-performance CPU or the like needs to be used in order to performcalculation processing at high speed, but in that case, problems such ashigh cost of the digital camera and an increase in power consumption mayarise.

An object of the present invention is to provide a technique capable ofreducing the cost and power consumption of an imaging apparatus byfacilitating detection of an incident angle of a light beam transmittedthrough a grating substrate.

The above and other objects and novel features of the present inventionwill become apparent from the description of the present specificationand the accompanying drawings.

Means for Solving the Problems

An outline of representative inventions disclosed in the presentapplication will be briefly described as follows.

That is, a typical imaging apparatus includes an image sensor, amodulator, and an image processing unit. The image sensor converts anoptical image captured by a plurality of pixels arranged in an array onan imaging surface into an image signal and outputs the converted imagesignal. The modulator is provided on a light receiving surface of theimage sensor and modulates intensity of light. The image processing unitperforms image processing on the image signal output from the imagesensor.

Moreover, the modulator includes a grating substrate, a first gratingpattern, and a second grating pattern. The first grating pattern isformed on a first surface of the grating substrate arranged in proximityto the light receiving surface of the image sensor. The second gratingpattern is formed on a second surface facing the first surface.

Each of the first grating pattern and the second grating pattern isconstituted of a plurality of concentric circles. The modulator performsintensity modulation on light transmitted through the second gratingpattern by the first grating pattern and outputs the modulated light tothe image sensor.

In particular, the plurality of concentric circles in each of the firstgrating pattern and the second grating pattern are formed from aplurality of concentric circles in which a pitch of the concentriccircles becomes finer in inverse proportion relative to a referencecoordinate serving as a center of the concentric circles. The pitch ofthe plurality of concentric circles becomes finer in inverse proportionrelative to the reference coordinate serving as the center of theconcentric circles.

Effects of the Invention

Effects obtained by a representative invention among the inventionsdisclosed in the present application will be briefly described asfollows.

(1) Processing time before obtaining an object image can be shortened.

(2) Hardware cost of the imaging apparatus can be reduced.

BRIEF DESCRIPTIONS OF THE DRAWINGS

FIG. 1 is an explanatory diagram showing an example of a configurationin an imaging apparatus according to a first embodiment;

FIG. 2 is an explanatory diagram showing an example of photographing bythe imaging apparatus in FIG. 1;

FIG. 3 is a flowchart showing an outline of image processing by an imageprocessing circuit included in the imaging apparatus in FIG. 1;

FIG. 4 is an explanatory diagram showing an example of in-plane shift ofa projection image from a front surface to a back surface of adouble-sided grating substrate by an oblique incident parallel ray;

FIG. 5 is a schematic diagram for describing generation of a moirefringe and a frequency spectrum in a case where the axes of the gratingpatterns of the double-sided grating substrate are aligned;

FIG. 6 is an explanatory diagram showing an example of a double-sidedgrating substrate formed by shifting axes of a grating pattern on thefront surface side and a grating pattern on the back surface side;

FIG. 7 is a schematic diagram for describing generation of a moirefringe and a frequency spectrum in a case where the grating patterns arearranged so as to be shifted from each other;

FIG. 8 is an explanatory diagram showing a calculation result of aspatial frequency spectral image when being irradiated with a total often light rays including a normal incident plane wave and other nineplane waves with different incident angles;

FIG. 9 is a bird's-eye view showing a calculation result of a spatialfrequency spectral image when being irradiated with a total of ten lightrays including a normal incident plane wave and other nine plane waveswith different incident angles;

FIG. 10 is an explanatory diagram for describing an angle formed by alight ray from each of points constituting an object with respect to animage sensor;

FIG. 11 is an explanatory diagram showing an example of a spatialfrequency spectrum in a case where the grating patterns are mutuallyshifted in a horizontal direction;

FIG. 12 is an explanatory diagram showing an example of a spatialfrequency spectrum in a case where the grating patterns are mutuallyshifted in a vertical direction;

FIG. 13 is an explanatory diagram showing an example of a configurationof an imaging apparatus according to a third embodiment;

FIG. 14 is a flowchart showing an outline of image processing by animage processing circuit included in the imaging apparatus in FIG. 13;

FIG. 15 is an explanatory diagram showing that projection of a gratingpattern on the front surface side to the back surface is enlarged morethan the original grating pattern in a case where the object to beimaged is at a finite distance;

FIG. 16 is an explanatory diagram showing an example of a configurationof an imaging apparatus according to a fourth embodiment;

FIG. 17 is an explanatory diagram showing an example of a configurationof a double-sided grating substrate according to a fifth embodiment; and

FIG. 18 is an external view showing an example of a portable informationterminal according to a sixth embodiment.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

In the embodiments described below, the invention will be described in aplurality of sections or embodiments when required as a matter ofconvenience. However, these sections or embodiments are not irrelevantto each other unless otherwise stated, and the one relates to the entireor a part of the other as a modification example, details, or asupplementary explanation thereof.

Also, in the embodiments described below, when referring to the numberof elements (including number of pieces, values, amount, range, and thelike), the number of the elements is not limited to a specific numberunless otherwise stated or except the case where the number isapparently limited to a specific number in principle, and the numberlarger or smaller than the specified number is also applicable.

Further, in the embodiments described below, it goes without saying thatthe components (including element steps) are not always indispensableunless otherwise stated or except the case where the components areapparently indispensable in principle.

Similarly, in the embodiments described below, when the shape of thecomponents, positional relation thereof, and the like are mentioned, thesubstantially approximate and similar shapes and the like are includedtherein unless otherwise stated or except the case where it isconceivable that they are apparently excluded in principle. The samegoes for the numerical value and the range described above.

Also, components having the same function are denoted by the samereference characters in principle throughout the drawings for describingthe embodiments, and the repetitive description thereof is omitted.Hereinafter, embodiments will be described in detail.

First Embodiment

<Configuration Example of Imaging Apparatus>

FIG. 1 is an explanatory diagram showing an example of a configurationof an imaging apparatus 101 according to a first embodiment.

The imaging apparatus 101 is configured to obtain an image of anexternal object without using an imaging lens and includes a modulator102, an image sensor 103, and an image processing circuit 106 as shownin FIG. 1.

The modulator 102 is fixed in close contact with a light receivingsurface of the image sensor 103, and has a configuration in which eachof grating patterns 104 and 105 is formed on a grating substrate 102 a.The grating substrate 102 a is made of a transparent material such asglass or plastic.

In the modulator 102, the grating pattern 104 serving as a secondgrating pattern is formed on a front surface of the grating substrate102 a. Also, the front surface of the grating substrate 102 a serves asa second surface. The grating pattern 104 includes concentric gratingpatterns in which an interval of the grating patterns, that is, a pitchof the grating patterns becomes narrower in inverse proportion to aradius from a center to an outer side.

In addition, the grating pattern 105 serving as a first grating patternis formed on a back surface of the grating substrate 102 a, that is, thesurface on the side in contact with the light receiving surface of theimage sensor 103. The back surface of the grating substrate 102 a servesas a first surface.

Similarly to the grating pattern 104, the grating pattern 105 alsoincludes concentric grating patterns in which a pitch of the gratingpatterns becomes narrower in inverse proportion to the radius from thecenter to the outer side.

The grating pattern 104 and the grating pattern 105 are formed by, forexample, depositing aluminum or the like by the sputtering method or thelike used for a semiconductor process. Shading is given by a patternhaving aluminum deposited therein and a pattern having no aluminumdeposited therein.

Note that formation of the grating patterns 104 and 105 is not limitedto this, and the grating patterns 104 and 105 may be formed by givingthe shading by, for example, printing with an inkjet printer or thelike.

The intensity of the light transmitted through the grating patterns 104and 105 is modulated by each of the grating patterns. The transmittedlight is received by the image sensor 103. The image sensor 103 isconstituted of, for example, a CCD (Charge Coupled Device) image sensor,a CMOS (Complementary Metal Oxide Semiconductor) image sensor, or thelike.

On the front surface of the image sensor 103, pixels 103 a serving aslight receiving elements are regularly arranged in a form of grating.The image sensor 103 converts an optical image received by the pixel 103a into an image signal which is an electric signal. The image signaloutput from the image sensor 103 is subjected to image processing by theimage processing circuit 106 serving as an image processing unit, and isthen output to a monitor display 107 or the like.

<Example of Photographing by Imaging Apparatus>

FIG. 2 is an explanatory diagram showing an example of photographing bythe imaging apparatus 101 in FIG. 1. FIG. 2 shows an example in which asubject 301 is photographed by the imaging apparatus 101 and isdisplayed on the monitor display 107.

As shown in the figure, photographing of the subject 301 is performed ina state where one surface of the modulator 102, namely, the frontsurface of the grating substrate 102 a on which the grating pattern 104is formed directly faces the subject 301.

<Example of Image Processing of Image Processing Circuit>

Subsequently, an outline of image processing by the image processingcircuit 106 will be described.

FIG. 3 is a flowchart showing an outline of the image processing by theimage processing circuit 106 included in the imaging apparatus 101 inFIG. 1.

First, for a moire fringe image output from the image sensor 103,two-dimensional FFT (Fast Fourier Transform) operation is performed foreach of color RGB (Red Green Blue) components, thereby obtaining afrequency spectrum (step S101).

Subsequently, frequency data on one side of the frequency spectrumobtained by the processing of the step S101 is extracted (step S102),and the intensity of the frequency spectrum is calculated (step S103),thereby obtaining an image.

Then, after noise removal processing is performed on the obtained image(step S104), contrast emphasis processing (step S105) or the like isperformed. Thereafter, the color balance of the image is adjusted (stepS106) and the image is output as a photographed image.

In the manner described above, the image processing by the imageprocessing circuit 106 is completed.

<Photographing Principle of Imaging Apparatus>

Subsequently, a photographing principle of the imaging apparatus 101will be described.

First, the concentric grating patterns 104 and 105 in which the pitchbecomes finer in inverse proportion to the radius from the center aredefined as follows. Now the case where a spherical wave close to a planewave and a plane wave used as reference light are caused to interferewith each other in a laser interferometer or the like is assumed.

When the radius from a reference coordinate serving as the center of theconcentric circles is r and a phase of the spherical wave at that pointis ϕ(r), the phase of the spherical wave can be represented by theExpression (1) below by using a coefficient β which determines themagnitude of the curvature of a wavefront.

[Expression 1]

ϕ(r)=βr ²  (1)

The reason why the phase is represented by the square of the radius r inspite of a spherical wave is that the wave is a spherical wave close toa plane wave, and thus approximation is possible solely by the lowestorder of expansion. When plane waves are caused to interfere with thelight having this phase distribution, the intensity distribution of theinterference fringes represented by the Expression (2) is obtained.

$\begin{matrix}\lbrack {{Expression}\mspace{14mu} 2} \rbrack & \; \\\begin{matrix}{{I(r)} = {\frac{1}{4}{{{\exp \; i\; {\varphi (r)}} + 1}}^{2}}} \\{= {\frac{1}{2}( {1 + {\cos \; \varphi}} )}} \\{= {\frac{1}{2}( {1 + {\cos \; \beta \; r^{2}}} )}}\end{matrix} & (2)\end{matrix}$

This corresponds to a fringe of concentric circles having a bright lineat a radial position that satisfies the Expression (3)

[Expression 3]

ϕ(r)=βr ²=2nπ (n=0,1,2, . . . )  (3)

When the pitch of the fringe is p, the following Expression (4) isobtained, and it can be seen that the pitch becomes narrower in inverseproportion to the radius.

$\begin{matrix}\lbrack {{Expression}\mspace{14mu} 4} \rbrack & \; \\{{{p\frac{d}{dr}{\varphi (r)}} = {{2p\; \beta \; r} = {2\pi}}}{{p(r)} = \frac{\pi}{\beta \; r}}} & (4)\end{matrix}$

Such a fringe is referred to as Fresnel zone plate. The grating patternhaving a transmittance distribution proportional to the intensitydistribution defined in this manner is used as the grating patterns 104and 105 shown in FIG. 1.

Now the case where a parallel ray is incident at an angle θ0 on themodulator 102 with a thickness t having such a grating pattern formed onboth surfaces as shown in FIG. 4 is assumed. When a refraction angle inthe modulator 102 is θ, light multiplied by the transmittance of thegrating of the front surface is incident on the back surface while beingshifted by δ=t·tan θ in terms of geometrical optics, and if the centersof the two concentric gratings are formed to be aligned with each other,the transmittance of the grating of the back surface is multiplied whilebeing shifted by δ.

At this time, intensity distribution as represented by the followingExpression (5) is obtained.

$\begin{matrix}{\mspace{79mu} \lbrack {{Expression}\mspace{14mu} 5} \rbrack} & \; \\\begin{matrix}{{{I( {x,y} )}{I( {{x + \delta},y} )}} = {\frac{1}{4}\{ {1 + {\cos \; {\beta ( {x^{2} + y^{2}} )}}} \} \{ {1 + {\cos \; {\beta ( {( {x + \delta} )^{2} + y^{2}} )}}} \}}} \\{= {\frac{1}{8}\{ {2 + {4\; \cos \; {\beta ( {r^{2} + {\delta \; x}} )}\cos \; \delta \; \beta \; x} +} }} \\ {{\cos \; 2{\beta ( {r^{2} + {\delta \; x}} )}} + {\cos \; 2\; \beta \; \delta \; x}} \}\end{matrix} & (5)\end{matrix}$

It can be seen that the fourth term of this expansion expression formsfringe patterns, which are equally spaced and straight in a direction ofthe shift of the two gratings, over a whole overlapped region. Thefringes generated at a relatively low spatial frequency due to theoverlapping of the fringes are referred to as moire fringes.

These equally spaced and straight fringes generate a sharp peak in thespatial frequency distribution obtained by two-dimensional Fouriertransform of the detected image. A value of δ, that is, an incidentangle θ of the light beam can be obtained from the frequency value.

It is obvious that such equally spaced moire fringes uniformly obtainedon the whole surface are generated at the same pitch regardless of thedirection of shift because of the symmetry of the concentric gratingarrangement. These fringes can be obtained because the grating patternis formed of the Fresnel zone plate, and it is considered that it is notpossible to obtain uniform fringes on a whole surface by other gratingpatterns.

It can be seen that the second term also generates fringes in which theintensity of the Fresnel zone plate is modulated by moire fringes. Inthis case, however, since the frequency spectrum of the product of thetwo fringes is a convolution of each of Fourier spectra, no sharp peakis obtained.

When a component having a sharp peak is selectively extracted from theExpression (5) as shown in the Expression (6), the Fourier spectrumthereof is as shown in the Expression (7).

$\begin{matrix}\lbrack {{Expression}\mspace{14mu} 6} \rbrack & \; \\{{I( {x,y} )} = {\frac{1}{8}( {2 + {\cos \; 2\; \delta \; \beta \; x}} )}} & (6) \\{\lbrack {{Expression}\mspace{14mu} 7} \rbrack \;} & \; \\\begin{matrix}{{F\lbrack {I( {x,y} )} \rbrack} = {\frac{1}{8}{F\lbrack {2 + {\cos \; 2\; \delta \; \beta \; x}} \rbrack}}} \\{= {{\frac{1}{4}{\delta ( {u,v} )}} + {\frac{1}{8}{\delta ( {{u + \frac{\delta \; \beta}{\pi}},v} )}} + {\frac{1}{8}{\delta ( {{u - \frac{\delta \; \beta}{\pi}},v} )}}}}\end{matrix} & (7)\end{matrix}$

Note that F represents Fourier transform operation, u and v are spatialfrequency coordinates in the x direction and the y direction, and δ withparentheses is a delta function. The result indicates that the peak ofthe spatial frequency of moire fringes is generated at a position ofu=±δβ/π in the spatial frequency spectrum of the detected image.

This state is shown in FIG. 5. In FIG. 5, layout diagrams of the lightbeam and the modulator 102, diagrams of the moire fringes, and schematicdiagrams of the spatial frequency spectrum are shown in the order fromthe left to the right. Specifically, FIG. 5(a) shows a case of normalincidence, FIG. 5(b) shows a case where a light beam is incident fromthe left side at an angle θ, and FIG. 5(c) shows a case where a lightbeam is incident from the right side at the angle θ.

The grating pattern 104 formed on the front surface side of themodulator 102 and the grating pattern 105 formed on the back surfaceside are coaxially arranged. In FIG. 5(a), since the shadow of thegrating pattern 104 and the shadow of the grating pattern 105 coincidewith each other, moire fringes are not generated.

In FIGS. 5(b) and 5(c), the same moire is generated because the shift ofthe grating pattern 104 and the grating pattern 105 is equal, and peakpositions of the spatial frequency spectrum also coincide with eachother. As a result, it is not possible to determine whether the incidentangle of the light beam is as shown in FIG. 5(c), from the spatialfrequency spectrum.

In order to avoid this, it is necessary to arrange the two gratingpatterns 104 and 105 so as to be shifted relative to the optical axisbeforehand as shown in FIG. 6 such that the shadows of the two gratingpatterns are overlapped so as to be shifted with respect to the lightbeam perpendicularly incident on the modulator 102.

When the relative shift of the shadows of the two gratings with respectto the normal incident plane wave on the axis is δ0, a shift δ caused bythe plane wave of the incident angle θ can be represented as theExpression (8).

[Expression 8]

δ=δ₀ +t tan θ  (8)

At this time, the peak of the spatial frequency spectrum of moirefringes of the light beam of the incident angle θ is at a positionrepresented by the Expression (9) on the positive side of the frequency.

$\begin{matrix}\lbrack {{Expression}\mspace{14mu} 9} \rbrack & \; \\{u = {\frac{\delta \; \beta}{\pi} = {\frac{1}{\pi}( {\delta_{0} + {t\; \tan \; \theta}} )\beta}}} & (9)\end{matrix}$

When the size of the image sensor is S and the number of pixels of theimage sensor in each of the x direction and the y direction is N, thespatial frequency spectrum of a discrete image calculated by fastFourier transform (FFT) is obtained in a range of −N/(2S) to +N/(2S).

Accordingly, considering that light is received equally at the incidentangle on the positive side and the incident angle on the negative side,the spectral peak position of the moire fringe by the normal incidentplane wave (θ=0) is reasonably determined to be located at a centralposition between an origin (DC: direct current component) position and afrequency position at the positive (+) side end, that is, a spatialfrequency position represented by the Expression (10).

$\begin{matrix}\lbrack {{Expression}\mspace{14mu} 10} \rbrack & \; \\{{\frac{1}{n}\delta_{0}\beta} = \frac{N}{4S}} & (10)\end{matrix}$

Therefore, the relative center position shift between the two gratingsis reasonably determined as represented by the Expression (11).

$\begin{matrix}\lbrack {{Expression}\mspace{14mu} 11} \rbrack & \; \\{\delta_{0} = \frac{\pi \; N}{4\; \beta \; S}} & (11)\end{matrix}$

FIG. 7 is a schematic diagram for describing generation of a moirefringe and a frequency spectrum in a case where the grating patterns 104and 105 are arranged so as to be shifted from each other.

Similarly to FIG. 5, layout diagrams of the light beam and the modulator102 are shown on the left side, diagrams of moire fringes are shown atthe middle, and schematic diagrams of the spatial frequency spectrum areshown on the right side. In addition, FIG. 7(a) is a case where thelight beam is a normal incident beam, FIG. 7(b) shows a case where thelight beam is incident from the left side at an angle θ, and FIG. 7(c)is a case where the light beam is incident from the right side at theangle θ.

The grating pattern 104 and the grating pattern 105 are arranged to beshifted from each other by δ0 in advance. Therefore, moire fringes aregenerated also in FIG. 7(a), and a peak appears in the spatial frequencyspectrum.

As described above, the shift amount δ0 is set such that a peak positionappears at a center of the spectral range on one side from the origin.At this time, since the shift δ increases in FIG. 7(b) and the shift δdecreases in FIG. 7(c), a difference between FIG. 7(b) and FIG. 7(c) canbe determined from the peak position of the spectrum unlike FIG. 5.

The spectral image of this peak is a bright point indicating a lightflux at infinity, which is nothing but a photographed image by theimaging apparatus 101 in FIG. 1.

When the maximum incident angle of the parallel ray that can be receivedis θmax, the maximum angle of view that can be received by the imagingapparatus 101 is given by the Expression (13) on the basis of theExpression (12).

$\begin{matrix}\lbrack {{Expression}\mspace{14mu} 12} \rbrack & \; \\{u_{\max} = {{\frac{1}{\pi}( {\delta_{0} + {t\; \tan \; \theta_{\max}}} )\; \beta} = \frac{N}{2S}}} & (12) \\\lbrack {{Expression}\mspace{14mu} 13} \rbrack & \; \\{{\tan \; \theta_{\max}} = \frac{\pi \; N}{4\; t\; \beta \; S}} & (13)\end{matrix}$

By analogy with the conventional image formation using a lens, in a casewhere a parallel ray with the angle of view of θmax is received byfocusing at the end of the image sensor, the effective focal length ofthe imaging apparatus 101 using no lens can be considered to correspondto the Expression (14).

$\begin{matrix}\lbrack {{Expression}\mspace{14mu} 14} \rbrack & \; \\{f_{eff} = {\frac{S}{2\; \tan \; \theta_{\max}} = \frac{2\; \beta \; t\; S^{2}}{\pi \; N}}} & (14)\end{matrix}$

As indicated by the expression (2), the transmittance distribution ofthe grating pattern fundamentally assumes a sinusoidal characteristic,and if such a component is present as a fundamental frequency componentof the grating pattern, it is also conceivable to enhance thetransmittance by binarizing the transmittance of the grating pattern andchanging the duty between the grating region having high transmittanceand the grating region having low transmittance so as to expand thewidth of the high transmittance region.

Although incident light beam has just one simultaneous incident angle inthe above description, it is necessary to assume a case where light rayshaving a plurality of incident angles are simultaneously incident, inorder to cause the imaging apparatus 101 to actually operate as acamera.

Such light rays having a plurality of incident angles would cause aplurality of images of the front side grating to be overlapped at thetime of the incidence on the grating pattern on the back surface side.If these mutually generate moire fringes, there would be a concern ofgenerating noise that might hinder detection of moire fringes with thegrating pattern 105 which is a signal component.

In practice, however, the overlapping of the images of the gratingpattern 104 generates no peak of the moire image, and solely theoverlapping with the grating pattern 105 on the back surface sidegenerates the peak.

The reason will be described below.

First, a large difference is that the overlapping of the shadows of thegrating pattern 104 on the front surface side by the light beams at aplurality of incident angles is not a product but a sum. In theoverlapping of the shadow of the grating pattern 104 by the light at oneincident angle and the grating pattern 105, the light intensitydistribution after transmission through the grating pattern 105 on theback surface side is obtained by multiplying the intensity distributionof the light corresponding to the shadow of the grating pattern 104 bythe transmittance of the grating pattern 105.

In contrast, since the overlapping of the shadows caused by a pluralityof light rays having different incident angles to be incident on thegrating pattern 104 on the front surface side is the overlapping of thelight rays, it is not a product but a sum. In the case of sum, thedistribution obtained by multiplying the distribution of the grating ofthe original Fresnel zone plate by the distribution of moire fringes isgenerated as represented in the Expression (15).

$\begin{matrix}\lbrack {{Expression}\mspace{14mu} 15} \rbrack & \; \\\begin{matrix}{{{I( {x,y} )} + {I( {{x + \delta},y} )}} = {{\frac{1}{2}\{ {1 + {\cos \; {\beta ( {x^{2} + y^{2}} )}}} \}} +}} \\{{\frac{1}{2}\{ {1 + {\cos \; {\beta ( {( {x + \delta} )^{2} + y^{2}} )}}} \}}} \\{= {1 + {\cos \; {\beta ( {r^{2} + {\delta \; x}} )}\cos \; \delta \; \beta \; x}}}\end{matrix} & (15)\end{matrix}$

Accordingly, the frequency spectrum thereof is represented by theoverlap integral of each of the frequency spectra. Therefore, even whenthe single moire spectrum has a sharp peak, merely ghosting of thefrequency spectrum of the Fresnel zone plate would occur at thatposition in practice. Namely, no sharp peak is generated in thespectrum.

Accordingly, the spectrum of the moire image to be detected even whenthe light rays having a plurality of incident angles are incident isconstantly the moire of the product of the grating pattern 104 on thefront surface side and the grating pattern 105 on the back surface sidealone, and the number of peaks of the spectrum to be detected is solelyone for one incident angle as long as the grating pattern 105 is single.

<Confirmation of Photographing Principle>

Hereinafter, results of a simulation performed to confirm the principleare shown in FIGS. 8 and 9. FIG. 8 is an explanatory diagram showing acalculation result of a spatial frequency spectral image when beingirradiated with a total of ten light rays including a normal incidentplane wave and other nine plane waves with different incident angles.FIG. 9 is a bird's-eye view showing a calculation result of a spatialfrequency spectral image when being irradiated with a total of ten lightrays including a normal incident plane wave and other nine plane waveswith different incident angles.

Both figures show a spectrum in a case where a total of 10 plane wavesincluding a normal incident plane wave, incident light with θx=50° andθy=300, incident light with θx=−30° and θy=70°, incident light withθx=10° and θy=−20°, incident light with θx=20° and θy=300, incidentlight with θx=300 and θy=−40°, incident light with θx=−10° and θy=400,incident light with θx=−20° and θy=−300, incident light with θx=−30° andθy=0°, and incident light with θx=400 and θy=500 are incident under theconditions that the sensor size of the image sensor 103 is 20 mm square,the viewing angle θmax=±700, grating coefficient β on the incident sideand emission side=50 (rad/mm2), δ0=0.8 mm, the number of pixels is1024×1024, the substrate thickness at the modulator 102 is 1 mm, and thesubstrate refractive index is 1.5.

FIG. 8 is a black and white inverted image of the spectral image, andFIG. 9 is a bird's-eye view showing the luminance of the spectral image.The original moire image itself is omitted because the grating pitch istoo thin to be visually recognized even when it is displayed in thedrawing of the present specification.

In the figure, the whole area of a spatial frequency spectral region inwhich the center indicates a DC component and the periphery indicates±N/2S is displayed. The DC component has a large value and is thusremoved by masking, and solely the peak component to be detected isdisplayed. Furthermore, since the original peak width of the spectrum istoo narrow to be visually recognized, contrast is emphasized.

In addition, in FIG. 8, the positions of the signal peaks are indicatedby open circles surrounding the positions. In the bird's-eye view ofFIG. 9, since the drawn line does not pass through the peak and thuscannot be displayed as it is, a result obtained by applying an averagingfilter of the mesh size is displayed.

Both figures basically show that 10 peaks can be detected as a total of20 peaks on both of positive and negative sides across the origin. Inthis case, the pitch of the outermost periphery of the grating patternis about 6 μm and the effective focal length is about 12.4 mm.

Now, the correspondence between the parallel ray to be detected whichhas been described so far and the light from the actual object will bedescribed schematically with reference to FIG. 10.

FIG. 10 is an explanatory diagram for describing an angle formed by alight ray from each of points constituting an object with respect to animage sensor.

Strictly speaking, the rays from respective points constituting thesubject 301 are incident as spherical waves from a point light source onthe modulator 102 and the image sensor 103 (hereinafter referred to as agrating sensor integrated substrate in FIG. 10) of the imaging apparatus101 in FIG. 1.

At this time, in a case where the grating sensor integrated substrate issufficiently small with respect to the subject 301 or is sufficientlydistant from the subject 301, the incident angles of light raysilluminating the grating sensor integrated substrate from the respectivepoints can be regarded to be the same.

On the basis of a relationship that a spatial frequency displacement Δuof the moire with respect to a minute angular displacement Δθ obtainedfrom the expression (9) is not more than 1/S which is the minimumresolution of the spatial frequency of the image sensor, the conditionfor determining Δθ as a parallel ray is represented by the Expression(16).

$\begin{matrix}\lbrack {{Expression}\mspace{14mu} 16} \rbrack & \; \\{{{\Delta \; u} = {{\frac{1}{\pi}\beta \; t\; \Delta \; \theta} \leq \frac{1}{S}}}{{\Delta \; \theta} \leq \frac{\pi}{s\; \beta \; t}}} & (16)\end{matrix}$

Accordingly, if Δθ<0.180, this condition can be realized when a sensorsize is 20 mm and a distance from the subject is 6 m.

By analogy with the above results, it can be seen that the imagingapparatus of the present invention is capable of the image formation ofan object at infinity.

As described above, an image of an external object can be obtained bysimple operation such as fast Fourier transform (FFT). Accordingly, itis possible to shorten the processing time before obtaining an objectimage.

Moreover, since there is no need to use a high-performance calculationprocessing apparatus, it is possible to reduce the hardware cost of theimaging apparatus 101. Furthermore, since the calculation processingtime is shortened, it is possible to reduce the power consumption of theimaging apparatus 101.

Second Embodiment

<Overview>

Although the case where an image output from the imaging apparatus 101is vertically long has been described in the first embodiment above, thecase where the output image is horizontally long will be described inthe second embodiment.

In the first embodiment, as described above, the grating pattern 104 andthe grating pattern 105 are formed so as to be shifted from each otherin the x direction (horizontal direction) of the image sensor 103. Inother words, the grating pattern 104 and the grating pattern 105 areformed so as to be shifted in a long side direction of a rectangularimage output from the image processing circuit 106.

<Example of Formation of Grating Pattern>

FIG. 11 is an explanatory diagram showing an example of a spatialfrequency spectrum in a case where the grating patterns 104 and 105 aremutually shifted in the horizontal direction.

At this time, the shape of the image sensor 103 is assumed to be asquare, and a pixel pitch thereof is equal in both of the x directionand the y direction. In this case, as shown on the right side of FIG.11, in the spatial frequency spectrum of the output of the image sensor,the image is reproduced to be separated to the left and right sideswithin the frequency range of ±N/S in both of the x direction and the ydirection.

In the example shown in FIG. 11, however, the image is basically limitedto a vertically long area. Generally, an image obtained by a digitalcamera or the like is a horizontally long rectangle with an aspect ratioof, for example, 3:2 or 4:3. Accordingly, as an arrangement of thegrating patterns 104 and 105 suitable for the horizontally longrectangle, an example shown in FIG. 12 is desirable.

FIG. 12 is an explanatory diagram showing an example of a spatialfrequency spectrum in a case where the grating patterns 104 and 105 aremutually shifted in the vertical direction.

As shown in FIG. 12, the grating pattern 104 and the grating pattern 105are formed so as to be shifted in the vertical direction of the imagesensor, that is, in the y direction of the image sensor. In other words,the grating pattern 104 and the grating pattern 105 are formed so as tobe mutually shifted in a short side direction of a rectangular imageoutput from the image processing circuit 106. As a result, the imageformed in the spatial frequency space of the image sensor output isvertically separated as shown on the right side of FIG. 12.

As described above, it is possible to make the image output from theimaging apparatus 101 horizontally long. Therefore, since an image canbe obtained in the same manner as a general digital camera, versatilityof the imaging apparatus 101 can be enhanced.

Third Embodiment

<Overview>

In the modulator 102 according to the first and second embodiments, thegrating pattern 104 and the grating pattern 105 with the same shape areformed on the front surface and the back surface of the gratingsubstrate 102 a, respectively, so as to be shifted from each other, sothat an image is formed by detecting the angles of the incident parallelrays from the spatial frequency spectrum of the moire fringes.

The grating pattern 105 on the back surface side is an optical elementthat comes in close contact with the image sensor 103 and modulates theintensity of incident light. Therefore, by setting the sensitivity ofthe image sensor effectively by taking into account the transmittance ofthe grating pattern 105 on the back surface side, it is possible tovirtually generate moire in the processed image.

<Configuration Example of Imaging Apparatus>

FIG. 13 is an explanatory diagram showing an example of a configurationof the imaging apparatus 101 according to a third embodiment.

The imaging apparatus 101 in FIG. 13 is different from the imagingapparatus 101 in FIG. 1 according to the first embodiment in that thegrating pattern 105 shown in FIG. 1 is not formed on the back surfaceside of the grating substrate 102 a. The other configurations aresimilar to those in FIG. 1, and thus the description thereof will beomitted.

By adopting the configuration shown in FIG. 13, it is possible toeliminate one grating pattern to be formed on the grating substrate 102a. This makes it possible to reduce the manufacturing cost of themodulator 102.

In this case, however, a pitch of the pixels 103 a of the image sensor103 needs to be fine enough to sufficiently reproduce the pitch of thegrating pattern, or a pitch of the grating pattern needs to be roughenough to be reproduced by the pixel pitch of the image sensor 103.

In the case where a grating pattern is to be formed on both surfaces ofthe grating substrate 102 a, it is not always necessary that the pitchof the grating pattern is resolvable by the pixel 103 a of the imagesensor 103, and it is only necessary that solely the moire image thereofis resolvable. Therefore, the pitch of the grating pattern can bedetermined independently of the pixel pitch.

However, in the case where the grating pattern is reproduced by theimage sensor 103, the grating pattern and the resolution of the imagesensor 103 need to be equal to each other. Therefore, an intensitymodulation circuit 106 c corresponding to the grating pattern 105(FIG. 1) on the back surface side for generating moire for the outputimage of the image sensor 103 is provided in the image processingcircuit 106.

<Example of Image Processing of Image Processing Circuit>

FIG. 14 is a flowchart showing an outline of image processing by theimage processing circuit 106 included in the imaging apparatus 101 inFIG. 13.

The flowchart in FIG. 14 is different from the flowchart in FIG. 3 ofthe first embodiment in the processing of step S201. In the processingof step S201, a moire fringe image corresponding to a grating pattern onthe back surface side is generated for the image output from the imagesensor 103 by the above-described intensity modulation circuit 106 c.

Thereafter, the processing of steps S202 to S208 of FIG. 14 is similarto the processing of steps S101 to S107 in FIG. 3 of the firstembodiment, and thus the description thereof will be omitted.

By providing the intensity modulation circuit 106 c in this manner, itis possible to obtain an effect similar to the case where the gratingpattern 105 on the back surface side (FIG. 1) is made variable, so thatthe detection light needs not necessarily be a parallel ray.

<Focusing>

FIG. 15 is an explanatory diagram showing that projection of the gratingpattern 104 on the front surface side to the back surface is enlargedmore than the grating pattern 104 in a case where the object to beimaged is at a finite distance.

As shown in FIG. 15, in a case where the grating pattern 104 on thefront surface side is irradiated with a spherical wave from a point 1301constituting an object and a shadow 1302 thereof is projected on a lowersurface, the image projected on the lower surface is substantiallyuniformly enlarged.

Therefore, equally spaced straight moire fringes would not be generatedby simply multiplying the transmittance distribution of the gratingpattern on the back surface side (corresponding to the grating pattern105 in FIG. 1) designed for the parallel ray. However, if the grating ofthe lower surface is enlarged in accordance with the uniformly enlargedshadow of the grating pattern 104 on the front surface side, it ispossible to generate equally spaced straight moire fringes again for theenlarged shadow 1302.

Thus, it is possible to selectively reproduce the light from the point1301 at a distance which is not necessarily at infinity. As a result,focusing becomes possible, and photographing can be performed whilefocusing at an arbitrary position instead of the photographing atinfinity described in the first embodiment.

In the manner described above, it is possible to enhance the convenienceof the imaging apparatus 101.

Fourth Embodiment

In a fourth embodiment, a technique for making the grating pattern 104on the front surface side of FIG. 1 variable will be described.

<Configuration Example and Operation Example of Imaging Apparatus>

FIG. 16 is an explanatory diagram showing an example of a configurationof an imaging apparatus 101 according to the fourth embodiment.

The imaging apparatus 101 in FIG. 16 is different from the imagingapparatus 101 in FIG. 1 according to the first embodiment in that aliquid crystal unit 108 is newly provided in the modulator 102 and afocus position designation input unit 109 is newly provided. Note thatthe configuration of the image sensor 103 in FIG. 16 is similar to theconfiguration in FIG. 1, and thus the description thereof will beomitted.

The liquid crystal unit 108 has a configuration in which a liquidcrystal layer (not shown) is provided on a glass substrate (not shown)on which a transparent electrode or the like is formed, and the liquidcrystal layer is formed so as to be sandwiched between the glasssubstrate and the grating substrate 102 a.

An arbitrary grating pattern 1403 is displayed on the liquid crystallayer, and the grating pattern 1403 serves as the grating pattern 104 onthe front surface side. Similarly to the case of FIG. 1, the gratingpattern 105 is formed on the back surface side of the grating substrate102 a in the modulator 102.

The focus position designation input unit 109 is an input unit whichsets a focus position corresponding to information such as the distanceto the subject, and is connected to the image processing circuit 106. Inaddition, the image processing circuit 106 includes a liquid crystaldrive circuit 106 a and a grating pattern generation circuit 106 b.

The grating pattern generation circuit 106 b generates a grating patternoptimum for focusing on the basis of the focus position input from thefocus position designation input unit 109. The liquid crystal drivecircuit 106 a performs display control by applying a voltage to thetransparent electrode formed on the glass substrate such that thegrating pattern generated by the grating pattern generation circuit 106b is displayed on the liquid crystal layer of the liquid crystal unit108.

Since the light from the finite distance point 1301 which is basicallycloser than infinity is divergent light, the same size as the gratingpattern 105 on the back surface side can be achieved on the back surfaceby displaying the grating pattern 104 on the front surface side so as tobe slightly smaller than the grating pattern 105.

In the manner described above, it is possible to achieve fasterfocusing.

Fifth Embodiment

In a fifth embodiment, another example of the grating pattern on thefront surface side formed on the double-sided grating substrate will bedescribed.

<Example of Formation of Grating Pattern>

FIG. 17 is an explanatory diagram showing an example of a configurationof the modulator 102 according to the fifth embodiment.

Although the case where the grating pattern 104 (FIG. 1) of themodulator 102 is formed by, for example, printing or the sputteringmethod has been described in the first embodiment, the grating patterncorresponding to the grating pattern 104 is constituted of a cylindricallens 110 in the modulator 102 in FIG. 17. Note that the grating pattern105 on the back surface side of the grating substrate 102 a is similarto that in FIG. 1 of the first embodiment.

In this case, the cylindrical lens 110 is arranged on the front surfaceof the grating substrate 102 a in the modulator 102 so as to form apattern similar to that of the grating pattern 104 in FIG. 1. Thecylindrical lens 110 is a lens formed with a cylindrical surface, and ithas a curvature of a convex lens in the vertical direction and nocurvature in the horizontal direction.

By forming the grating pattern with the cylindrical lens 110 in thismanner, it is possible to greatly reduce the loss of light quantity. Forexample, when a grating pattern in which shading is given by a printedpattern or the like is used as described in the first embodiment, theprinted portion of the grating pattern blocks light, leading to a largeloss of light quantity.

In contrast, light is not blocked in the case of the cylindrical lens110, and it is thus possible to enhance light use efficiency.

As a result, the S/N ratio (Signal-to-Noise ratio) in the imagingapparatus 101 can be increased, and it is thus possible to enhance thedrawing performance.

Sixth Embodiment

<Configuration Example of Portable Information Terminal>

In a sixth embodiment, a portable information terminal configured byusing the imaging apparatus 101 according to the fifth embodiment willbe described.

FIG. 18 is an external view showing an example of a portable informationterminal 200 according to the sixth embodiment.

For example, the portable information terminal 200 is a smartphone orthe like. Note that the portable information terminal 200 is not limitedto a smartphone, and may be a portable terminal such as a tabletincluding a built-in camera.

The portable information terminal 200 incorporates the imaging apparatus101. An aperture window 202 is provided on the back surface of theportable information terminal 200, and the modulator 102 shown in FIG.16 is provided in the portable information terminal 200 so as to come inproximity to the aperture window 202.

In addition, a focus adjustment knob 201 is provided on a side surfaceof one long side of the portable information terminal 200. The knob 201corresponds to the focus position designation input unit 109 in thefourth embodiment.

The focus position is set by turning the knob 201, and an arbitrarygrating pattern 1403 is displayed on the liquid crystal layer of theliquid crystal unit 108 in FIG. 16 in accordance with the set focusposition. As a result, it is possible to photograph an image of anobject at an arbitrary distance.

The imaging apparatus 101 is capable of increasing the effective focallength in accordance with the Expression (14) shown in the firstembodiment, and this makes it possible to enlarge an aperture whilekeeping the imaging apparatus 101 thin.

In the case of a general digital camera for smartphones using a lens, itis inevitable to downsize the aperture of the lens in order to reducethe thickness of the information portable device. Therefore, the focallength becomes short, with the result that the image is flattened andblur cannot be produced in the out-of-focus part of the image.

In contrast, the imaging apparatus 101 is capable of enlarging theaperture as described above, and it is thus possible to produce theaesthetic quality of the blur in the image.

In the manner described above, it is possible to realize the portableinformation terminal 200 with enhanced drawing performance.

In the foregoing, the invention made by the inventors of the presentinvention has been concretely described based on the embodiments.However, it is needless to say that the present invention is not limitedto the foregoing embodiments and various modifications and alterationscan be made within the scope of the present invention.

Note that the present invention is not limited to the embodimentsdescribed above, and includes various modification examples. Forexample, the above embodiments have been described in detail in order tomake the present invention easily understood, and the present inventionis not necessarily limited to those having all the describedconfigurations.

Also, a part of the configuration of one embodiment may be replaced withthe configuration of another embodiment, and the configuration of oneembodiment may be added to the configuration of another embodiment.Furthermore, another configuration may be added to a part of theconfiguration of each embodiment, and a part of the configuration ofeach embodiment may be eliminated or replaced with anotherconfiguration.

REFERENCE SIGNS LIST

-   101 Imaging apparatus-   102 Modulator-   102 a Grating substrate-   103 Image sensor-   103 a Pixel-   104 Grating pattern-   105 Grating pattern-   106 Image processing circuit-   106 a Liquid crystal drive circuit-   106 b Grating pattern generation circuit-   106 c Intensity modulation circuit-   107 Monitor display-   108 Liquid crystal unit-   109 Focus position designation input unit-   110 Cylindrical lens-   200 Portable information terminal-   201 Knob-   202 Aperture window

1. An imaging apparatus comprising: an image sensor configured toconvert an optical image captured by a plurality of pixels arranged inan array on an imaging surface into an image signal and output theconverted image signal; a modulator provided on a light receivingsurface of the image sensor and configured to modulate intensity oflight; and an image processing unit configured to perform imageprocessing of the image signal output from the image sensor, wherein themodulator includes: a grating substrate; a first grating pattern formedon a first surface of the grating substrate arranged in proximity to thelight receiving surface of the image sensor; and a second gratingpattern formed on a second surface facing the first surface, each of thefirst grating pattern and the second grating pattern is constituted of aplurality of concentric circles, and the modulator performs intensitymodulation on light transmitted through the second grating pattern bythe first grating pattern and outputs the modulated light to the imagesensor.
 2. The imaging apparatus according to claim 1, wherein a pitchof the plurality of concentric circles in each of the first gratingpattern and the second grating pattern becomes finer in inverseproportion relative to a reference coordinate serving as a center of theconcentric circles.
 3. The imaging apparatus according to claim 1,wherein the image processing unit calculates a frequency spectrum byperforming two-dimensional Fourier transform on the image signal outputfrom the image sensor.
 4. The imaging apparatus according to claim 2,wherein the reference coordinate of the first grating pattern and thereference coordinate of the second grating pattern are different inposition.
 5. The imaging apparatus according to claim 4, wherein thereference coordinate of the first grating pattern and the referencecoordinate of the second grating pattern are shifted from each other ina short side direction of an image output from the image processingunit.
 6. The imaging apparatus according to claim 1, wherein the secondgrating pattern is formed of a cylindrical lens.
 7. An imaging apparatuscomprising: an image sensor configured to convert an optical imagecaptured by a plurality of pixels arranged in an array on an imagingsurface into an image signal and output the converted image signal; amodulator provided on a light receiving surface of the image sensor andconfigured to modulate intensity of light; and an image processing unitconfigured to perform image processing of the image signal output fromthe image sensor, wherein the modulator includes: a grating substrate;and a second grating pattern formed on a second surface facing a firstsurface of the grating substrate arranged in proximity to the lightreceiving surface of the image sensor, the second grating pattern isconstituted of a plurality of concentric circles, and the imageprocessing unit generates a moire fringe image obtained by adding moireto the image signal output from the image sensor.
 8. The imagingapparatus according to claim 7, wherein a pitch of the plurality ofconcentric circles in the second grating pattern becomes finer ininverse proportion relative to a reference coordinate serving as acenter of the concentric circles.
 9. The imaging apparatus according toclaim 7, wherein the image processing unit calculates a frequencyspectrum by performing two-dimensional Fourier transform on thegenerated moire fringe image.
 10. The imaging apparatus according toclaim 7, wherein the second grating pattern is formed of a cylindricallens.
 11. An imaging apparatus comprising: an image sensor configured toconvert an optical image captured by a plurality of pixels arranged inan array on an imaging surface into an image signal and output theconverted image signal; a modulator configured to modulate intensity oflight; and an image processing unit configured to perform imageprocessing of the image signal output from the image sensor, wherein themodulator includes: a grating substrate; a first grating pattern formedon a first surface of the grating substrate arranged in proximity to alight receiving surface of the image sensor; and a grating patterndisplay unit formed on a second surface facing the first surface andconfigured to display a second grating pattern on the second surface,each of the first grating pattern formed on the first surface of thegrating substrate and the second grating pattern displayed on thegrating pattern display unit is constituted of a plurality of concentriccircles, and the grating pattern display unit displays the secondgrating pattern under control of the image processing unit.
 12. Theimaging apparatus according to claim 11, wherein a pitch of theplurality of concentric circles in each of the first grating pattern andthe second grating pattern becomes finer in inverse proportion relativeto a reference coordinate serving as a center of the concentric circles.13. The imaging apparatus according to claim 11, wherein the referencecoordinate of the second grating pattern displayed by the gratingpattern display unit and the reference coordinate of the first gratingpattern formed on the first surface are different in position.
 14. Theimaging apparatus according to claim 11, further comprising: a focusposition designation input unit configured to designate a focus positionof a subject, wherein the image processing unit changes a shape of theplurality of concentric circles displayed by the grating pattern displayunit based on focus position information designated by the focusposition designation input unit.
 15. The imaging apparatus according toclaim 10, wherein the image processing unit calculates a frequencyspectrum by performing two-dimensional Fourier transform on the imagesignal output from the image sensor.